Telco LLMs Application: Use Cases Development
Introduction
As we promote projects related to Large Language Models (LLMs) with various departments in the company, we encounter individuals who either have excessive expectations about the LLMs' inference capabilities and potential, or conversely, have a critical perspective due to issues like hallucination and accuracy limitations. While LLMs are powerful and useful tools, they are not a silver bullet that can solve all problems. Therefore, discussing the omnipotence or pessimism regarding LLMs is pointless; we should focus on utilizing LLMs in a tailored manner for specific problems and requirements.
Clear use cases provide the purpose and direction for LLM development. Since last year, we have actively engaged various departments to contemplate specific use cases to leverage LLMs to their full potential, identifying approximately more than 100 potential use cases. We are starting to apply Telco LLMs in key domains such as contact centers and network management.
Contact Center
Telcos invest significant resources in their customer centers to manage customer satisfaction and provide prompt responses, making this a major pain point for telecom companies. Customer service representatives are required to resolve customer issues promptly amidst a flood of incoming calls. Especially for inexperienced representatives, additional searching and learning are needed when faced with complex problems. After the call, they have to document the service record and perform follow-up actions. In the worst-case scenario, they might have to handle tasks from the previous call while attending to the next one. Post-call activities can account for up to 40% of the total call handling duties.
If we can reduce these repetitive tasks and save the representatives' time, they can focus on high-value tasks such as customer analysis and sales activities. This can bring additional revenue to the telco and provide incentives for the representatives. By leveraging LLM, we can specifically provide a representative assistant to support around 20 different tasks. Here are some examples:
Realtime-call tasks:
Analyzing customer intent
Searching for service policies and relevant products
Post-call tasks:
Summarizing key points of the call
Categorizing the call type
Classifying the product discussed
Extracting to-do items
Analyzing customer sentiment
When we applied a general LLM to various tasks performed by service representatives, a survey of the representatives showed a satisfaction level of only around 40%. This is because the LLM lacks the knowledge and terminology specific to Telco products and policies. To improve task accuracy, we are continuously training the Telco LLM by establishing instructional data for specific tasks in collaboration with the representatives, based on conversations between representatives and customers. Additionally, we are incorporating regular representative tests and feedback data, and we plan to implement this in SKT's contact center by 2024
Network Management
Network management and operations are the core foundational tasks for Telco's infrastructure. Network quality (throughput, latency, stability, etc.) directly impacts customer satisfaction, making network management one of the key competitive advantages for Telcos. Managing and operating the network requires high SLA (Service Level Agreement) standards for all systems to ensure network QoS (Quality of Service) and prevent failures. Therefore, a cautious, step-by-step approach is needed when applying LLM to network management.
The Telco network consists of heterogeneous and complex networks including wireless networks, transport networks, core networks, and IP networks, and there are various tasks to which LLM can be applied. These tasks include serving as a knowledge assistant to help network operators, providing operator training, script writing, querying equipment databases, generating automatic reports, classifying logs, monitoring equipment, sending alerts, and performing root cause analysis.
We are developing an operator assistant, which is the task with the lowest risk. However, we found that using a general LLM resulted in an RAG accuracy of below 60%, which does not meet the commercial standard. As mentioned earlier, the Telco network requires extensive and complex domain knowledge. Therefore, we first need to narrow down the target network area and integrate the scattered and complex knowledge into a unified database. Next, we must build training datasets for the Telco LLM and embedding model to enhance QA (Question Answering) performance. This includes training on specialized knowledge, equipment manuals, standard documents, incident response documents, and terminology. Building this data and knowledge database requires a significant amount of time and effort.
What is Needed for Effective LLM Use Cases?
Key points must be emphasized for the prerequisites and successful commercialization of the above-mentioned use case projects. Close cooperation between domain experts and AI experts is essential. However, differences in working styles or cultures, varying levels of understanding of the technology, and conflicting interests often make it difficult even to start the projects. Frequent iterations of data collection, construction, and LLM model training are necessary, but bridging the gap with business departments accustomed to a clear schedule-driven waterfall approach and an ROI perspective is not easy. To achieve what can be called "AI-like" results, we must collectively overcome the "non-AI-like" process challenges.
To overcome these difficulties, it is crucial to set clear goals, establish effective communication, ensure efficient data collection and management, optimize resource allocation, and secure internal momentum for the initiatives.